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Background 

Policy makers and researchers are not only interested in the program’s main effects (“what 
works’”’), but also moderation effects (“work for whom, under what conditions’). A critical step 
in designing Cluster Randomized Trials (CRTs) to detect these effects is conducting a priori 
statistical power analyses. Statistical power for main effects in CRTs has an extensive research 
base and user-friendly computer programs for executing power analyses (e.g., Optimal Design, 
[Raudenbush, Spybrook, Congdon, Liu, & Martinez, 2011], CRT-Power [Borenstein & Hedges, 
2012], and PowerUp! [Dong & Maynard, 2013]). However, research on power analysis to detect 
moderator effects in CRTs is very limited although more recently, some advancements have been 
made in this area (Bloom, 2005; Bloom & Spybrook, 2015; Jaciw, 2014; Mathieu, Aguinis, 
Culpepper, & Chen, 2012; Spybrook, Kelcey, & Dong, 2016). 


Purpose 

The purpose of this paper is to present results of recent advances in power analyses to detect the 
moderator effects in CRTs. This paper focus on demonstration of the software PowerUp!- 
Moderator (Dong, Kelcey, Spybrook, & Maynard, 2016), which was based on recent work 
(Dong, Kelcey, & Spybrook, 2016; Dong, Spybrook, & Kelcey, 2016; Spybrook, Kelcey, & 
Dong, 2016). This paper provides a resource for researchers seeking to design CRTs with 
adequate power to detect the moderator effects of the programs. 


Methods 

We implemented the closed-form power formulas that we previously derived and validated 
through simulations in an easy to use interactive Excel spreadsheet PowerUp!-Moderator. We 
first outline the software PowerUp!-Moderator. We then delineate the factors that are necessary 
for power analysis of moderator effects. Finally we demonstrate how to calculate the minimum 
detectable effect size difference (MDESD) and power using several examples. 


Results 

Table 1 covers the models and corresponding worksheets for calculation of minimum detectable 
effect size difference (MDESD) and power of moderator effects in two- and three-level cluster 
randomized trials (CRTs) (1.e., cluster random assignment designs[CRA]). Column 1 indicates 
the number of total levels of clustering (2 for two-level CRTs/CRA). Column 2 indicates the 
model number, e.g., CRA2-1N is for the model with a level-1 moderator (Colum 3) with 
nonrandomly varying slope (Column 4), CRA2-1R is for the model with a level-1 moderator 
(Colum 3) with random slope (Column 4), CRA2-2 is for the model with a level-2 moderator 
(Colum 3). Columns 5 and 6 contain the worksheet labels for the calculation of MDESD and 
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power for the binary moderators, while Columns 7 and 8 contain the worksheet labels for the 
calculation of MDESD and power for the continuous moderators. The users can click these 
worksheet labels to go to the corresponding worksheets to conduct power analyses. 


As in the power analysis of the main treatment effect, the power of the moderator effect in two- 
level CRTs is associated with the Type I error rate (a), one-tailed or two-tailed test, the 
unconditional intraclass correlation (ICC), the proportion of clusters in the treatment group (P), 
the proportion of variance explained by covariates, and the sample sizes for clusters and 
individuals. In addition, if the moderator is a binary variable, the power is also associated with 
the proportion of sample in one moderator subgroup; if the moderator is at level-1 with a random 
slope, the power is also associated with the effect heterogeneity (@) for the level-1 moderator 
across level-2 units. We demonstrate the calculation of MDESD and power using several 
examples below. 


Suppose a team of researchers are designing a two-level CRT to test the efficacy of a school- 
based intervention on student achievement. They are interested in student-level moderator effects 
and school-level moderator effects. The moderator can be a binary or continuous variable. They 
approach the moderator power analyses from two perspectives: (1) what is the MDESD given 
power of 0.80 and (2) what is the power for an effect size of 0.20. Table 2 shows the results of 
MDESD and power for the total numbers (/) of schools of 40 and 80 under some assumptions 
(In the presentation we will share resources with participants concerning the identification and 
selection of practical values of design parameters within a few different substantive examples.). 


We have the following findings from Table 2. First, a design always has a smaller MDESD, or 
larger power for a fixed effect size when the level-2 sample size is bigger (e.g., J = 80 vs. 40). 
Second, the MDESD is larger, or the power is smaller for a fixed effect size when the moderator 
is at school level compared to the student level. Third, under these assumptions, when the 
moderator is at student level, the nonrandomly varying moderator slope model has a smaller 
MDESD, or bigger power for a fixed effect size than the random moderator slope model. Finally, 
the MDESD as defined by Cohen’s d for the binary moderator is always twice the value of the 
MDESD defined by the standardized coefficient for the continuous moderator when the 
moderator is school level or the moderator is at student level with the nonrandomly varying 
slope. 


Tables 3, 4, and 5 below demonstrate the calculation of MDESD and power for three examples 
under the same assumptions in Table 2 using the software that we developed. Table 4 
demonstrates the calculation of MDESD regarding Cohen’s d and 95% confidence interval for a 
binary level-1 moderator with nonrandomly varying slope. Table 5 demonstrates the calculation 
of MDESD regarding Cohen’s d and 95% confidence interval for a continuous level- 1 moderator 
with random slope. Table 6 demonstrates the calculation of power for a binary level-2 moderator. 
In all worksheets, the user needs to input the assumptions about the design parameters (e.g., ICC 
and R’) and significance test (e.g., alpha level and one/two-tailed test) that are highlighted in 
yellow, and the results of the MDESD and its confidence interval, or power will be calculated 
automatically. 
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Tables & Figures 


Table 1. PowerUp!-Moderator to Detect Moderator Effects in 2- and 3-Level CRTs: Models and Corresponding 


PowerUp!-Moderator to Detect Moderator Effects in 2- and 3-Level CRTs: Models and Corresponding Worksheets 


1 2 3 4 5 6 7 8 
Number of Model Level of Slope of Lower Binary Moderator Continuous Moderator 
Total Levels of Numb Mod Level 
Study Design Clustering — Gdeeiior Moderator MDESD Calculation | Power Calculation) MDESD Calculation | Power Calculation 
Cluster Random Assignment Designs (Level of Assignment ~ Level of Analysis) 
Simple Cluster 
P CRA2-IN 1 Nonrandomly | Gra? 1N MDESD | CRA2_1N Power | CRA2_ INC MDESD | CRA2_1NC Power 
Random Varying 
Assignment 2 
(CRA), or CRA2-1R 1 Random CRA2_1R_MDESD | CRA2_1R Power | CRA2_ IRC MDESD | CRA2_1RC Power 
Cluster CRA2-2 2 NA CRA2_2_ MDESD CRA2_2 Power CRA2_2C_ MDESD CRA2_2C_ Power 
Randomized N an 
Trials (CRTs) CRA3-IN 1 Via Y | CRA3_IN_MDESD | CRA3_IN Power] CRA3_INC_MDESD | CRA3_1NC_Power 
CRA3-1R 1 Random CRA3_1R_MDESD | CRA3_1R_ Power | CRA3_IRC_ MDESD | CRA3_1RC_ Power 
3 CRA3-2N 2) ee CRA3_2N_MDESD | CRA3_2N Power| CRA3_2NC_MDESD | CRA3_2NC_ Power 
CRA3-2R 2 Random CRA3_2R_ MDESD | CRA3_2R_ Power | CRA3_2RC MDESD | CRA3_2RC Power 
CRA3-3 3 NA CRA3_3_ MDESD CRA3_3_ Power CRA3_3C_MDESD CRA3_3C_Power 
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Table 2. MDESD and statistical power of two-level CRTs 


MDESD Power 
Level of Slope of Lower Level Binary Continuous Binary Continuous 
Moderator Moderator Moderator Moderator Moderator Moderator 
J=40 J=80 J=40 J=80 J=40 J=80 J=40 J=80 
1 Nonrandomly Varying 0.11 0.08 0.06 0.04 1.00 1.00 1.00 1.00 
1 Random 0.26 0.18 0.25 0.17 0.56 0.86 0.63 0.91 
2 NA 0.67 0.45 0.34 0.23 0.13 0.24 0.39 0.70 


Note. Under the assumptions: n = 100, p = 0.23, R; =0.5, R> =0.5, Rj, =0, @ =0.3, P= 0.5, Q = 0.5, power = 0.8 for the 
calculation of MDESD, and effect size difference = 0.2 for the calculation of power, a two-sided test with a = 0.05. 
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Table 3: MDESD and 95% confidence interval for a binary level-1 moderator with nonrandomly varying slope in a two-level CRT 


Model CRA2-1N: MDESD Calculator for Two-Level Cluster Random Assignment Design — Treatment at Level 2 and 
Binary Moderator at Level 1 (Nonrandomly varying moderator slope model) 


Assumptions Comments 

Alpha Level («) 0.05 Probability of a Type I error 

Two-tailed or One-tailed Test? y 

Power (1-8) 0.80 Statistical power (1-probability of a Type II error) 

Rho (ICC) Proportion of variance in outcome that is between clusters 

P Proportion of Level 2 units randomized to treatment: Jy; / Jr +Jo) 

Q Proportion of Level 1 units in Moderator subgroup: n,; / (n; + no) 

R, Proportion of variance in Level 1 outcomes explained by Level 1 covariates 
oe Number of Level 1 covariates excluding the moderator 


n (Average Cluster Size) 


Mean number of Level 1 units per Level 2 cluster (harmonic mean recommended) 


J (Sample Size [# of Clusters]) 


Number of Level 2 units 


T, (Power) 


M (Multiplier) 2.80 Computed from T, and T; 
T, (Precision) 1.96 Determined from alpha level, given two-tailed or one-tailed test 


| 0.84 [Determined from given power level 
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Table 4: MDESD and 95% confidence interval for a continuous level-1 moderator with random slope in a two-level CRT 


Model CRA2-1RC: MDESD Calculator for Two-Level Cluster Random Assignment Design — Treatment at Level 2 and Continuous 
Moderator at Level 1 (Random moderator slope model) 


Assumptions Comments 

Alpha Level («) 0.05 Probability of a Type I error 

Two-tailed or One-tailed Test? 2 

Power (1-8) 0.80 Statistical power (1-probability of a Type II error) 

Rho (ICC) Proportion of variance in outcome that is between clusters 
The effect heterogeneity for the level-1 moderator across clusters in the model that is not 

a) conditional on treatment variable, which is the proportion of the variance between clusters on the 
effect of the moderator to the between-cluster residual variance. @=T,/T%o 

P Proportion of Level 2 units randomized to treatment: J+ / J++ Jc) 

Re Proportion of variance in Level 1 outcomes explained by Level 1 covariates 

Ro Proportion of variance between Level-2 clusters on the effect of Level-1 moderator explained by 


level-2 predictors. 


n (Average Cluster Size) 


Mean number of Level 1 units per Level 2 cluster (harmonic mean recommended) 


Sample Size [# of Clusters 
P 


Number of Level 2 units 


M (Multiplier) 


2.88 Computed from T, and T, 


T, (Precision) 


2.02 Determined from alpha level, given two-tailed or one-tailed test 


T> (Power) 


0.85 Determined from given power level 
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Table 5: Power for a binary level-2 moderator in a two-level CRT 


Model CRA2-2: Power Calculator for Two-Level Cluster Random Assignment Design — Treatment at Level 2 and 
Binary Moderator at Level 2 

Assumptions Comments 

Alpha Level («) 0.05  |Probability of a Type I error 

Two-tailed or One-tailed Test? Z 

Effect Size Difference 0.2  |Effect Size Difference regarding Cohen's d. 

Rho (ICC) Proportion of variance in outcome that is between clusters 

P Proportion of Level 2 units randomized to treatment: Jy / J + Jo) 

Q Proportion of Level 2 units in Moderator subgroup: J; / (J: + Jo) 

Re Proportion of variance in Level 1 outcomes explained by Level 1 covariates 

RS Proportion of variance in Level 2 outcome explained by Level 2 covariates 

a Number of Level 2 covariates excluding the moderator and moderator*Treatment 
n (Average Cluster Size) Mean number of Level 1 units per Level 2 cluster (harmonic mean recommended) 
J Sample Size [# of Clusters]) Number of Level 2 units 

Noncentrality Parameter 1.26 |Automatically computed from the above assumptions 

Power (1-8) Statistical power (1-probability of a Type I error) 
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